Incorporating Visual Information into Sound Source Separation

We present a method of improving sound source separation using vision. The sound source separation is an essential function to accomplish auditory scene understanding by separating a stream of sounds generated from multiple sound sources. By separating a stream of sounds, recognition process, such as speech recognition, can simply work on a single stream, not mixed sound of several speakers. The performance is known to be improved by using binaural microphone and microphone array which provide spatial information for separation. However, these methods still have around 20 degree of positional ambiguities. In this paper, we further added visual information to provide more speci c and accurate position information. As a result, separation capability was drastically improved. We argue, from the experiments, in this paper, that integration of vision and auditory sensory inputs improves cognitive tasks such as auditory stream separation.